Bootstrapping by Felix Bittmann

Author:Felix Bittmann
Language: eng
Format: epub
Publisher: De Gruyter Oldenbourg
Published: 2021-05-10T09:47:58.005000+00:00

Figure 4.2 Comparison of treatment and control group using kernel-density plots.

While the scores are on average higher in the treatment group, the distributions are clearly not normal and outliers with very high scores can be found in both groups. The means and standard deviations are 7.49 (4.69) in the control group and 9.07 (4.20) in the treatment group. The point estimates of the arithmetic means differ, but are these differences significant? Many people would conduct a simple t-test here, despite being aware of the fact that the underlying distributions should be normal. After all, how bad could it be if everyone else does it? We proceed with the two-sided t-test and receive a p-value of 0.0092, which is below the critical value of 0.01. Just to be sure, we additionally compute Welch’s approximation, which gives us an even lower p-value of 0.0064. We could easily just stop here, conclude that the drug works, and celebrate our success. Instead, we are left with a nagging feeling that something isn’t right, perhaps because we ignored a basic lesson from Stats 101. Eventually, we decide to use a permutation test. Since we are thorough and want to minimize the influence of the Monte Carlo error, we specify 75,000 resamples (an exhaustive test would be impossible even on modern hardware). The result is a p-value of 0.0119, which is above the critical value of 0.01. Worse yet, it is almost twice as high as the p-value reported by the Welch-test. So what is the correct conclusion? As others have pointed out, without a doubt, the result produced by the permutation test (Hesterberg, 2015). It is well-known that the t-test is unreliable when baseline distributions are skewed, as it is a clear violation of its basic assumptions (Sutton, 1993). Why are t-tests so popular, then? The reason appears to be tradition. The t-test was invented as an approximation for the permutation test about 100 years ago, when all calculations were done using pencil and paper. Given the massive computational power of modern hardware, however, why settle for second best? When you have worked on an important experiment for weeks or months, it seems fair to let your laptop take an extra half an hour to perform the calculations required for the permutation tests.

After this motivating example, let’s explore how permutation tests actually work. For a more thorough introduction to the topic, see Good (2013). The basic idea can be easily visualized when you have two groups, A and B, with sizes and , respectively (note that the group sizes can be unequal). Using these samples, you want to test whether a statistic of interest, in most cases the mean, differs between the groups. Suppose the means of both groups are denoted by and . Your null hypothesis is that the means are equal, while the alternative hypothesis specifies that they are unequal and that there is a significant difference:

(4.6)

(4.7)

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.